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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 



- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )E3 Responsive to communication(s) filed on 26 March 2007 . 
2a)D This action is FINAL. 2b)KI This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1, 453 O.G. 213. 

Disposition of Claims 

4) K Claim(s) 1-23 is/are pending in the application. 

4a) Of the above claim(s) 1^6 is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) KI Claim(s) 7-23 is/are rejected. 

7) Q Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) KI The drawing(s) filed on 27 August 2003 is/are: a)|EI accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)Q None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) ^ Information Disclosure Statement(s) (PTO/SB/08) 5 ) d Notice of Informal Patent Application 

Paper No(s)/Mail Date 1/20/04 . 6) □ Other: . 
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DETAILED ACTION 



Election/Restrictions 

1 . Claims 1 -6 are withdrawn from further consideration pursuant to 37 CFR 1 .142(b), as 
being drawn to a nonelected Group I, there being no allowable generic or linking claim. 
Applicant timely traversed the restriction (election) requirement in the reply filed on 3/26/07. 



Claim Rejections - 35 USC § 101 

2. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

The USPTO "Interim Guidelines for Examination of Patent Applications for Patent Subject 
Matter Eligibility" (Official Gazette notice of 22 November 2005), Annex IV, reads as 
follows: 

Descriptive material can be characterized as either "functional descriptive material" or "nonfunctional descriptive 
material." In this context, "functional descriptive material" consists of data structures and computer programs 
which impart functionality when employed as a computer component. (The definition of "data structure" is "a 
physical or logical relationship among data elements, designed to support specific data manipulation functions." 
The New IEEE Standard Dictionary of Electrical and Electronics Terms 308 (5th ed. 1993).) "Nonfunctional 
descriptive material" includes but is not limited to music, literary works and a compilation or mere arrangement 
of data. 

When functional descriptive material is recorded on some computer-readable medium it becomes structurally and 
functionally interrelated to the medium and will be statutory in most cases since use of technology permits the 
function of the descriptive material to be realized. Compare In re Lowry, 32 F.3d 1579, 1583-84, 32 USPQ2d 
1 03 1 , 1 035 (Fed. Cir. 1 994) (claim to data structure stored on a computer readable medium that increases 
computer efficiency held statutory) and Warmerdam, 33 F.3d at 1360-61, 31 USPQ2d at 1759 (claim to 
computer having a specific data structure stored in memory held statutory product-by-process claim) with 
Warmerdam, 33 F.3d at 1361, 31 USPQ2d at 1760 (claim to a data structure per se held nonstatutory). 

In contrast, a claimed computer-readable medium encoded with a computer program is a computer element 
which defines structural and functional interrelationships between the computer program and the rest of the 
computer which permit the computer program's functionality to be realized, and is thus statutory. See Lowry, 32 
F.3d at 1583-84, 32 USPQ2d at 1035. 
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Claim(s) 20-23 is/are rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter as follows. Claim 20 defines an "article comprising 
machine-readable storage medium containing instruction that if executed enable a system" 
embodying functional descriptive material. However, the claim does not define a computer- 
readable medium or memory and is thus non-statutory for that reason (i.e., "When functional 
descriptive material is recorded on some computer-readable medium it becomes structurally and 
functionally interrelated to the medium and will be statutory in most cases since use of 
technology permits the function of the descriptive material to be realized" - Guidelines Annex 
IV). The examiner suggests amending the claim to embody the program on "computer-readable 
medium" or equivalent in order to make the claim statutory. Any amendment to the claim 
should be commensurate with its corresponding disclosure. 

Claim Rejections - 35 USC § 102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

4. Claims 7, 8, 13, 15-17, 20 and 21 are rejected under 35 U.S.C. 102(b) as being 
anticipated by USPN 6,219,640 to Basu et al. "Basu". 



As to claim 7, Basu discloses a method comprising: 
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Recognizing a face of a subject from first entries in a database (column 6, lines 32-58, 
wherein the facial recognition of an individual speaking is performed); 

Recognizing audio- visual speech of the subject from second entries in the database 
(column 5, lines 28-42 and column 11, lines 10-31, wherein speech is recognized using the audio 
of a audio-visual signal and that speech is verified using the utterance verifier, which uses the 
video of the audio-visual signal); and 

Identifying the subject based on recognizing the face and recognizing the audio-visual 
speech (column 8, lines 43-50, wherein the recognition results determined by the face and audio- 
visual speech recognizers are combined to produce a joint score which is used to identify the 
individual). 

As to claim 8, Basu discloses the method of claim 7, further comprising providing the 
subject access to a restricted area after identifying the subject (column 11, lines 60-65). 

As to claim 13, Basu discloses the method of claim 7, wherein recognizing the audio- 
visual speech further comprises detecting and tracking a mouth region using vector machine 
classifiers (column 10, lines 14-25). 

As to claim 15, Basu discloses the method of claim 7, further comprising results of 
recognizing the face and recognizing the audio-visual speech pattern according to a 
predetermined weighting to identify the subject (column 8, lines 43-50). 
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As to claim 16, Basu discloses a system comprising: 

At least one capture device to capture audio-visual information from a subject (column 4, 
lines 15-23 and column 13, lines 31-67); 

A first storage device coupled to the at least one capture device to store code to enable 
the system (column 13, lines 31-67) to recognize a face of the subject from first entries in a 
database (column 6, lines 32-58, wherein the facial recognition of an individual speaking is 
performed), recognize audio-visual speech of the subject from second entries in the database 
(column 5, lines 28-42 and column 11, lines 10-31, wherein speech is recognized using the audio 
of a audio-visual signal and that speech is verified using the utterance verifier, which uses the 
video of the audio-visual signal), and identify the subject based on the face and the audio-speech 
(column 8, lines 43-50, wherein the recognition results determined by the face and audio-visual 
speech recognizers are combined to produce a joint score which is used to identify the 
individual); and 

A processor coupled to the first storage to execute the code (column 13, lines 3 1-67). 

As to claim 17, Basu discloses the system of claim 16, wherein the database is stored in 
the first storage device (column 13, lines 3 1-67 and column 3, lines 45-48). 

As to claim 20, please refer to the rejection of claim 7 above. 
As to claim 21 , please refer to the rejection of claim 8 above. 
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Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

6. Claims 9-1 1, 18 and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Basu in view of "Hidden Markov Models for Face Recognition" to Nefian et al. ("Nefian"). 

As to claim 9, Basu discloses the method of claim 7. 

Basu does not disclose expressly wherein recognizing the face comprises modeling an 
image including the face using an embedded hidden Markov model. 

However, Nefian discloses facial recognition comprising modeling an image including 
the face using an embedded hidden Markov model (Abstract). 

Basu & Nefian are combinable because they are from the same art of image processing, 
specifically facial recognition. 

At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to recognizing the face comprises modeling an image including the face using an 
embedded hidden Markov model, as taught by Nefian, with the facial recognition disclosed by 
Basu. 
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The suggestion/motivation for doing so would have been to reduce significantly the 
computational complexity of previous HMM-based face recognition system while preserving the 
same recognition rate (Nefian, Abstract and Introduction). 

Therefore, it would have been obvious to combine Basu with Nefian to obtain the 
invention as specified in claim 9. 

As to claim 10, the combination of Basu and Nefian disclose the method of claim 9, 
further comprising obtaining observation vectors from a sampling window of the image (Nefian, 
section 4, paragraphs 1 and 2). 

As to claim 1 1, the combination of Basu and Nefian disclose the method of claim 10, 
wherein the observation vectors comprise discrete cosine transform coefficients (Nefian, section 
4, paragraph 3). 

As to claim 1 8, please refer to the rejection of claim 9 above. 
As to claim 22, please refer to the rejection of claim 9 above. 

7. Claim 12 is rejected under 35 U.S.C. 103(a) as being unpatentable over Basu in view of 
USPN 7,130,446 to Rui et al. ("Rui"). 

As to claim 1 2, Basu discloses the method claim 7. 
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Basu does not disclose expressly wherein the recognizing the face comprises performing 
a Viterbi decoding algorithm. 

However, Rui discloses a facial recognition process that comprises performing a Viterbi 
decoding algorithm (column 19, line 65 - column 20, line 10). 

Basu & Rui are combinable because they are from the same art of image processing, 
specifically facial recognition. 

At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to include in the facial recognition process disclosed by Basu, performing a Viterbi 
decoding algorithm, as taught by Rui. 

The suggestion/motivation for doing so would have been to determine the best contour 
for contour selection using the Viterbi algorithm (Rui, column 19, line 65 - column 20, line 10). 

Therefore, it would have been obvious to combine Basu with Rui to obtain the invention 
as specified in claim 12. 

8. Claims 14, 19 and 23 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Basu in view of "Speaker Independent Audio-Visual Continuous Speech Recognition" to Liang 
et al. ("Liang"). 

As to claim 14, Basu discloses the method of claim 7. 

Basu does not disclose expressly wherein recognizing the audio-visual speech comprises 
modeling an image and an audio sample using a coupled hidden Markov model. 
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However, Liang discloses recognizing the audio-visual speech comprises modeling an 
image and an audio sample using a coupled hidden Markov model (Abstract). 

Basu & Liang are combinable because they are from the same art of image processing. 

At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to recognizing the audio-visual speech comprises modeling an image and an audio sample 
using a coupled hidden Markov model, as taught by Liang, with the process of recognizing 
audio-visual speech disclosed by Basu. 

The suggestion/motivation for doing so would have been model the audio and visual state 
asynchrony while preserving their natural correlation over time (Liang, Abstract). 

Therefore, it would have been obvious to combine Basu with Liang to obtain the 
invention as specified in claim 14. 

As to claim 19, please refer to the rejection of claim 14 above. 
As to claim 23, please refer to the rejection of claim 14 above. 

Conclusion 

9. The prior art made of record and not relied upon is considered pertinent to applicants 
disclosure. 

USPN 6,594,629 to Basu et al. discloses face and audio-visual speech recognition. 
US 2004/0267521 to Cutler et al. discloses face and audio-visual speech recognition. 
USPN 6,219,639 to Bakis et al. discloses face and audio- visual speech recognition. 
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USPN 7,133,535 to Huang et al. discloses modeling speech and facial images using 
HMMs. 

10. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Aaron W. Carter whose telephone number is (571) 272-7445. 
The examiner can normally be reached on 8am - 4:30 am (Mon. - Fit). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Bhavesh Mehta can be reached on (571) 272-7453. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 




:er 



AU 2624 



